{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "run_review_scorer.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python2",
      "display_name": "Python 2"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l9aTVeuq-azu",
        "colab_type": "text"
      },
      "source": [
        "# NLP Review Scorer (Current Version: v2)\n",
        "\n",
        "**Disclaimer: This is only a toy. You should seriously treat your rebuttal despite the what scores are given below. Wish you good luck with your paper submission!**\n",
        "\n",
        "I know some of you are thinking about how to convert paper review to a numerical score.\n",
        "Yes, the time has come.\n",
        "\n",
        "In this notebook, you will be able to convert your review to overall score (hopefully in range 1~5).\n",
        "\n",
        "I assume that you have followed the pre-steps on GitHub: https://github.com/ymcui/NLP-Review-Scorer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ulMElLJygSvl",
        "colab_type": "text"
      },
      "source": [
        "## IMPORTANT NOTE: I would suggest you to copy this notebook to YOUR Colab (and then it will be totally safe to run your own data), instead of directly running it here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23lw9AXX--mT",
        "colab_type": "text"
      },
      "source": [
        "## Step 1: Mount your Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FXE6JqBjCA-z",
        "colab_type": "code",
        "outputId": "5583cfdd-ab27-4de0-a189-e8cf91ddfef6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m6HeCrot_gD2",
        "colab_type": "text"
      },
      "source": [
        "## Step 2: Unzip model to Colab\n",
        "Note that, the model will be updated occasionally according to the prediction performance. I will only keep the latest model here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "18Yf8Bqk_uEX",
        "colab_type": "code",
        "outputId": "f2a37103-6ae6-4933-eadc-876b47fde37b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "source": [
        "!unzip -n /content/drive/My\\ Drive/review_model_v2.zip -d /content/bert\n",
        "!mv /content/bert/*/* /content/bert/"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  /content/drive/My Drive/review_model_v2.zip\n",
            "  inflating: /content/bert/model_v2/vocab.txt  \n",
            "  inflating: /content/bert/model_v2/model.ckpt-0.meta  \n",
            "  inflating: /content/bert/model_v2/bert_config.json  \n",
            "  inflating: /content/bert/model_v2/model.ckpt-0.index  \n",
            "  inflating: /content/bert/model_v2/model.ckpt-0.data-00000-of-00001  \n",
            "  inflating: /content/bert/model_v2/tokenization.py  \n",
            "  inflating: /content/bert/model_v2/modeling.py  \n",
            "  inflating: /content/bert/model_v2/optimization.py  \n",
            "  inflating: /content/bert/model_v2/run_classifier.py  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I8jQx-wIARWx",
        "colab_type": "text"
      },
      "source": [
        "## Step 3: Input your review and RUN!\n",
        "\n",
        "Note that, it is better to remove '\\n' in your review before copy to `review_text` field.\n",
        "\n",
        "Be careful not to remove quote marks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KvvM-JcowNMq",
        "colab_type": "code",
        "outputId": "cb6d7d7a-5014-400f-ee46-d52a697365f7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# -*- coding: utf-8 -*-\n",
        "\"\"\"run_squad_on_colab.ipynb\n",
        "\n",
        "Automatically generated by Colaboratory.\n",
        "\"\"\"\n",
        "\n",
        "import datetime\n",
        "import json\n",
        "import os\n",
        "import pprint\n",
        "import random\n",
        "import string\n",
        "import sys\n",
        "import tensorflow as tf\n",
        "\n",
        "'''\n",
        "assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'\n",
        "TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "print('TPU address is', TPU_ADDRESS)\n",
        "\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "with tf.Session(TPU_ADDRESS) as session:\n",
        "  print('TPU devices:')\n",
        "  pprint.pprint(session.list_devices())\n",
        "\n",
        "  # Upload credentials to TPU.\n",
        "  with open('/content/adc.json', 'r') as f:\n",
        "    auth_info = json.load(f)\n",
        "  tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n",
        "  # Now credentials are set for all future sessions on this TPU.\n",
        "'''\n",
        "\n",
        "\n",
        "\"\"\"### Prepare and import BERT modules\n",
        "With your environment configured, you can now prepare and import the BERT modules. The following step clones the source code from GitHub and import the modules from the source. Alternatively, you can install BERT using pip (!pip install bert-tensorflow).\n",
        "\"\"\"\n",
        "\n",
        "# import python modules defined by BERT\n",
        "import sys\n",
        "#global variable \n",
        "sys.path += ['bert']\n",
        "\n",
        "import collections\n",
        "import modeling\n",
        "import optimization\n",
        "import tokenization\n",
        "from run_classifier import ReviewProcessor, file_based_convert_examples_to_features, file_based_input_fn_builder, model_fn_builder, PaddingInputExample\n",
        "import numpy as np\n",
        "\n",
        "review_text = \"This is a very good paper, outstanding paper, brilliant paper. I have never seen such a good paper before. It was well-written and the models are novel. The evaluations are sound and the results achieve state-of-the-art performance. It should be definitely accepted or I will be angry.\" #@param {type:\"raw\"}\n",
        "review_sample= [\"EMNLP\",\"0\",\"0\",review_text]\n",
        "\n",
        "vocab_file='/content/bert/vocab.txt'\n",
        "bert_config_file='/content/bert/bert_config.json'\n",
        "init_checkpoint='/content/bert/model.ckpt-0'\n",
        "\n",
        "do_train=False\n",
        "do_predict=True #@param [\"False\", \"True\"] {type:\"raw\"}\n",
        "train_batch_size=32\n",
        "predict_batch_size=8\n",
        "eval_batch_size=8\n",
        "max_seq_length=512\n",
        "save_checkpoints_steps=5000\n",
        "do_lower_case=False\n",
        "use_tpu=False\n",
        "warmup_proportion=0.1\n",
        "learning_rate=1e-5\n",
        "num_train_epochs=1\n",
        "\n",
        "def main():\n",
        "  output_dir = '/content/result'\n",
        "  tf.gfile.MakeDirs(output_dir)\n",
        "  print('***** Model output directory: {} *****'.format(output_dir))\n",
        "\n",
        "  ########################################################################\n",
        "\n",
        "  tf.logging.set_verbosity(tf.logging.INFO)\n",
        "\n",
        "  bert_config = modeling.BertConfig.from_json_file(bert_config_file)\n",
        "  tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
        "\n",
        "  #validate_flags_or_throw(bert_config)\n",
        "  \n",
        "  processor = ReviewProcessor()\n",
        "\n",
        "  tf.gfile.MakeDirs(output_dir)\n",
        "  tpu_cluster_resolver = None\n",
        "  if use_tpu:\n",
        "    tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)\n",
        "  is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2\n",
        "  run_config = tf.contrib.tpu.RunConfig(\n",
        "      cluster=tpu_cluster_resolver,\n",
        "      model_dir=output_dir,\n",
        "      save_checkpoints_steps=5000,\n",
        "      keep_checkpoint_max=2,\n",
        "      tpu_config=tf.contrib.tpu.TPUConfig(\n",
        "          iterations_per_loop=1000,\n",
        "          num_shards=8,\n",
        "          per_host_input_for_training=is_per_host))\n",
        "\n",
        "\n",
        "  train_examples = None\n",
        "  num_train_steps = None\n",
        "  num_warmup_steps = None\n",
        "\n",
        "  model_fn = model_fn_builder(\n",
        "      bert_config=bert_config,\n",
        "      init_checkpoint=init_checkpoint,\n",
        "      learning_rate=learning_rate,\n",
        "      num_train_steps=num_train_steps,\n",
        "      num_warmup_steps=num_warmup_steps,\n",
        "      use_tpu=use_tpu,\n",
        "      use_one_hot_embeddings=use_tpu)\n",
        "\n",
        "  # If TPU is not available, this will fall back to normal Estimator on CPU\n",
        "  # or GPU.\n",
        "  estimator = tf.contrib.tpu.TPUEstimator(\n",
        "      use_tpu=use_tpu,\n",
        "      model_fn=model_fn,\n",
        "      config=run_config,\n",
        "      train_batch_size=train_batch_size,\n",
        "      eval_batch_size=eval_batch_size,\n",
        "      predict_batch_size=predict_batch_size)\n",
        "\n",
        "  if do_predict:\n",
        "    predict_examples = processor.get_single_examples([review_sample])\n",
        "    num_actual_predict_examples = len(predict_examples)\n",
        "    if use_tpu:\n",
        "      # TPU requires a fixed batch size for all batches, therefore the number\n",
        "      # of examples must be a multiple of the batch size, or else examples\n",
        "      # will get dropped. So we pad with fake examples which are ignored\n",
        "      # later on.\n",
        "      while len(predict_examples) % predict_batch_size != 0:\n",
        "        predict_examples.append(PaddingInputExample())\n",
        "\n",
        "    predict_file = os.path.join(output_dir, \"predict.tf_record\")\n",
        "    file_based_convert_examples_to_features(predict_examples,\n",
        "                                            max_seq_length, tokenizer,\n",
        "                                            predict_file)\n",
        "\n",
        "    tf.logging.info(\"***** Running prediction*****\")\n",
        "    tf.logging.info(\"  Num examples = %d (%d actual, %d padding)\",\n",
        "                    len(predict_examples), num_actual_predict_examples,\n",
        "                    len(predict_examples) - num_actual_predict_examples)\n",
        "    tf.logging.info(\"  Batch size = %d\", predict_batch_size)\n",
        "\n",
        "    predict_drop_remainder = True if use_tpu else False\n",
        "    predict_input_fn = file_based_input_fn_builder(\n",
        "        input_file=predict_file,\n",
        "        seq_length=max_seq_length,\n",
        "        is_training=False,\n",
        "        drop_remainder=predict_drop_remainder)\n",
        "\n",
        "    result = estimator.predict(input_fn=predict_input_fn)\n",
        "    tf.logging.info(result)\n",
        "    tf.logging.info(\"***** Predict results *****\")\n",
        "    for (i, prediction) in enumerate(result):     \n",
        "      probabilities = prediction[\"probabilities\"]\n",
        "      output_line = \"\\t\".join(\n",
        "            str(class_probability)\n",
        "            for class_probability in probabilities)\n",
        "      output_line = predict_examples[i].guid + \"\\t\" + output_line\n",
        "      break\n",
        "\n",
        "    new_review_text = '.\\n'.join(review_text.split('. '))\n",
        "    print(\"***********REVIEW**************\")\n",
        "    print(new_review_text)\n",
        "    print(\"***********SCORE***************\")\n",
        "    print(\"Paper\\tRecommendation\\tConfidence\")\n",
        "    print(output_line)\n",
        "    print(\"********************************\")\n",
        "\n",
        "if __name__ == '__main__':\n",
        "  main()\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0712 04:43:26.449141 140526333785984 deprecation_wrapper.py:119] From bert/optimization.py:87: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n",
            "W0712 04:43:26.453605 140526333785984 deprecation_wrapper.py:119] From bert/modeling.py:93: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "***** Model output directory: /content/result *****\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0712 04:43:27.447454 140526333785984 lazy_loader.py:50] \n",
            "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "  * https://github.com/tensorflow/io (for I/O related ops)\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "W0712 04:43:27.449174 140526333785984 estimator.py:1984] Estimator's model_fn (<function model_fn at 0x7fce82434de8>) includes params argument, but params are not passed to Estimator.\n",
            "I0712 04:43:27.451684 140526333785984 estimator.py:209] Using config: {'_save_checkpoints_secs': None, '_num_ps_replicas': 0, '_keep_checkpoint_max': 2, '_task_type': 'worker', '_global_id_in_cluster': 0, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fce824aec90>, '_model_dir': '/content/result', '_protocol': None, '_save_checkpoints_steps': 5000, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None, eval_training_input_configuration=2), '_tf_random_seed': None, '_save_summary_steps': 100, '_device_fn': None, '_cluster': None, '_experimental_distribute': None, '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': None, '_experimental_max_worker_delay_secs': None, '_evaluation_master': '', '_eval_distribute': None, '_train_distribute': None, '_master': ''}\n",
            "I0712 04:43:27.460908 140526333785984 tpu_context.py:209] _TPUContext: eval_on_tpu True\n",
            "W0712 04:43:27.462445 140526333785984 tpu_context.py:211] eval_on_tpu ignored because use_tpu is False.\n",
            "W0712 04:43:27.466645 140526333785984 deprecation_wrapper.py:119] From bert/run_classifier.py:349: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.\n",
            "\n",
            "W0712 04:43:27.468339 140526333785984 deprecation_wrapper.py:119] From bert/run_classifier.py:353: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n",
            "I0712 04:43:27.470549 140526333785984 run_classifier.py:353] Writing example 0 of 1\n",
            "I0712 04:43:27.477463 140526333785984 run_classifier.py:327] *** Example ***\n",
            "I0712 04:43:27.478394 140526333785984 run_classifier.py:328] guid: EMNLP\n",
            "I0712 04:43:27.480664 140526333785984 run_classifier.py:330] tokens: [CLS] This is a very good paper , outstanding paper , brilliant paper . I have never seen such a good paper before . It was well - written and the models are novel . The evaluation ##s are sound and the results achieve state - of - the - art performance . It should be definitely accepted or I will be angry . [SEP]\n",
            "I0712 04:43:27.482208 140526333785984 run_classifier.py:331] input_ids: 101 1188 1110 170 1304 1363 2526 117 6976 2526 117 8431 2526 119 146 1138 1309 1562 1216 170 1363 2526 1196 119 1135 1108 1218 118 1637 1105 1103 3584 1132 2281 119 1109 10540 1116 1132 1839 1105 1103 2686 5515 1352 118 1104 118 1103 118 1893 2099 119 1135 1431 1129 5397 3134 1137 146 1209 1129 4259 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "I0712 04:43:27.486104 140526333785984 run_classifier.py:332] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "I0712 04:43:27.487267 140526333785984 run_classifier.py:333] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "I0712 04:43:27.489196 140526333785984 run_classifier.py:334] label: 0.0 0.0 \n",
            "I0712 04:43:27.491328 140526333785984 <ipython-input-3-f643c4b23cf7>:141] ***** Running prediction*****\n",
            "I0712 04:43:27.493096 140526333785984 <ipython-input-3-f643c4b23cf7>:144]   Num examples = 1 (1 actual, 0 padding)\n",
            "I0712 04:43:27.494535 140526333785984 <ipython-input-3-f643c4b23cf7>:145]   Batch size = 8\n",
            "W0712 04:43:27.496659 140526333785984 deprecation_wrapper.py:119] From bert/run_classifier.py:383: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.\n",
            "\n",
            "I0712 04:43:27.498286 140526333785984 <ipython-input-3-f643c4b23cf7>:155] <generator object predict at 0x7fce824ad910>\n",
            "I0712 04:43:27.499986 140526333785984 <ipython-input-3-f643c4b23cf7>:156] ***** Predict results *****\n",
            "I0712 04:43:27.501566 140526333785984 estimator.py:612] Could not find trained model in model_dir: /content/result, running initialization to predict.\n",
            "W0712 04:43:27.533273 140526333785984 deprecation.py:323] From bert/run_classifier.py:419: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.data.experimental.map_and_batch(...)`.\n",
            "W0712 04:43:27.534221 140526333785984 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/data/python/ops/batching.py:273: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
            "W0712 04:43:27.537266 140526333785984 deprecation_wrapper.py:119] From bert/run_classifier.py:392: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.\n",
            "\n",
            "W0712 04:43:27.545195 140526333785984 deprecation.py:323] From bert/run_classifier.py:399: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.cast` instead.\n",
            "I0712 04:43:27.566498 140526333785984 estimator.py:1145] Calling model_fn.\n",
            "I0712 04:43:27.567423 140526333785984 tpu_estimator.py:2965] Running infer on CPU\n",
            "I0712 04:43:27.568826 140526333785984 run_classifier.py:495] *** Features ***\n",
            "I0712 04:43:27.570300 140526333785984 run_classifier.py:497]   name = input_ids, shape = (?, 512)\n",
            "I0712 04:43:27.572200 140526333785984 run_classifier.py:497]   name = input_mask, shape = (?, 512)\n",
            "I0712 04:43:27.577828 140526333785984 run_classifier.py:497]   name = is_real_example, shape = (?,)\n",
            "I0712 04:43:27.580545 140526333785984 run_classifier.py:497]   name = label_ids, shape = (?, 2)\n",
            "I0712 04:43:27.583003 140526333785984 run_classifier.py:497]   name = segment_ids, shape = (?, 512)\n",
            "W0712 04:43:27.591067 140526333785984 deprecation_wrapper.py:119] From bert/modeling.py:171: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
            "\n",
            "W0712 04:43:27.595274 140526333785984 deprecation_wrapper.py:119] From bert/modeling.py:409: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.\n",
            "\n",
            "W0712 04:43:27.627661 140526333785984 deprecation_wrapper.py:119] From bert/modeling.py:490: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.\n",
            "\n",
            "W0712 04:43:27.695137 140526333785984 deprecation.py:323] From bert/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use keras.layers.dense instead.\n",
            "W0712 04:43:32.818211 140526333785984 deprecation_wrapper.py:119] From bert/run_classifier.py:528: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.\n",
            "\n",
            "I0712 04:43:33.978528 140526333785984 run_classifier.py:530] **** Trainable Variables ****\n",
            "I0712 04:43:33.979645 140526333785984 run_classifier.py:536]   name = bert/embeddings/word_embeddings:0, shape = (28996, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.983639 140526333785984 run_classifier.py:536]   name = bert/embeddings/token_type_embeddings:0, shape = (2, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.986321 140526333785984 run_classifier.py:536]   name = bert/embeddings/position_embeddings:0, shape = (512, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.988571 140526333785984 run_classifier.py:536]   name = bert/embeddings/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.992289 140526333785984 run_classifier.py:536]   name = bert/embeddings/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.993941 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.995754 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.997384 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:33.999110 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.000626 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.002298 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.003994 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.005533 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.007163 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.008732 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.010361 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.012090 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.013537 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.015094 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.016721 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.018425 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.019906 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.021573 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.023257 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.024892 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.026473 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.028314 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.029798 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.031163 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.032680 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.033984 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.035574 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.037008 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.038574 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.040172 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.041959 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.043551 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.044873 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.046616 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.048429 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.049992 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.051450 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.052956 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.054758 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.056365 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.057735 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.059611 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.061212 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.062673 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.064299 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.065830 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.067523 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.068583 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.070084 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.072170 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.075726 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.077393 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.079408 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.080841 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.082447 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.083904 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.085994 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.087505 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.088884 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.090440 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.092215 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.093900 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.095596 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.097103 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.098855 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.100509 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.101984 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.103564 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.105221 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.107117 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.109618 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.111426 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.112744 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.114533 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.115845 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.117526 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.118949 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.120692 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.123840 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.126207 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.128506 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.131558 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.134377 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.136850 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.140218 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.142487 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.145546 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.148281 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.150609 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.154213 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.156618 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.159734 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.162470 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.164988 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.168354 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.171353 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.173629 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.176062 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.177206 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.179147 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.180116 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.181427 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.182378 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.183980 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.184786 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.185996 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.186820 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.188301 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.189400 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.190831 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.192835 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.194180 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.195276 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.196876 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.198683 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.200653 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.203201 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.205317 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.207653 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.210201 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.211566 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.213713 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.216759 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.218094 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.220313 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.222456 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.224297 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.226461 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.228616 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.231055 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.233089 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.235306 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.237673 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.239106 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.241267 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.243410 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.245472 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.247694 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.248939 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.252032 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.253350 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.255880 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.257988 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.260235 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.262367 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.264621 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.266809 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.268503 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.271662 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.273087 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.274115 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.277328 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.281783 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.283092 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.283807 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.286267 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.287293 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.290416 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.291397 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.292325 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.293260 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.296602 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.297461 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.299498 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.302604 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.303622 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.304608 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.305893 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.306602 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.308063 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.309407 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.310887 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.312587 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.313904 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.315382 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.316765 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.318093 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.320224 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.321050 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.322860 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.323873 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.325140 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.325967 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.326818 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.327853 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.328547 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.329359 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.330231 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.331192 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.332087 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.332865 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.333679 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.334436 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.335233 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.336024 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.336992 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.337609 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.338510 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.339615 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.340836 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.341542 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.342458 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.343652 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.344708 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.345412 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.346229 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.347074 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.347886 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_12/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.348887 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.349834 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.350846 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.351773 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.352750 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.353678 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.354686 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.355606 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.365482 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.366254 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.368243 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.369343 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.371181 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.372745 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.374191 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.375576 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_13/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.376926 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.378901 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.379723 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.381603 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.382678 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.384449 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.385900 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.387280 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.388739 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.390291 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.391777 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.393168 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.395025 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.396409 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.397829 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.399528 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_14/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.400671 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.402051 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.403546 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.405332 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.406999 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.408178 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.409576 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.410909 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.412353 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.413898 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.415150 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.416609 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.418379 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.419567 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.420882 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.422297 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_15/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.424154 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.425549 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.427071 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.428474 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.430026 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.432230 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.433211 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.434143 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.435025 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.436753 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.439435 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.440866 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.442409 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.443963 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.445498 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.447299 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_16/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.448726 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.450227 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.451793 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.453330 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.455358 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.456680 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.458112 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.459526 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.460901 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.462769 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.468677 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.471309 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.476032 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.477077 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.478790 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.480353 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_17/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.481731 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.483201 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.484564 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.486376 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.487415 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.489279 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.490346 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.491594 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.492882 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.494256 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.495570 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.497565 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.498709 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.500036 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.501351 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.502657 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_18/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.504065 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.506129 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.506850 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.508577 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.509655 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.511540 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.513040 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.514365 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.515863 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.517268 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.518645 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.519978 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.521842 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.523274 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.524694 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.526115 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_19/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.527467 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.528868 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.530183 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.531996 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.533680 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.534811 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.536818 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.538141 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.539788 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.540739 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.542454 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.544081 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.545459 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.546778 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.548064 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.549417 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_20/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.550820 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.552858 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.554126 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.555497 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.556874 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.558281 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.559623 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.561031 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.562815 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.564340 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.565757 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.567154 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.568506 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.570341 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.571851 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.573208 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_21/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.574609 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.576077 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.577395 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.578826 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.580319 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.582379 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.583565 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.585007 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.586473 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.587909 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.589379 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.590786 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.592699 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.593969 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.595252 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.596193 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_22/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.597522 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/self/query/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.598982 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/self/query/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.600460 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/self/key/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.602547 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/self/key/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.603729 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/self/value/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.605093 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/self/value/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.606468 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/output/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.607827 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.609211 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.610590 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/attention/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.612006 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/intermediate/dense/kernel:0, shape = (1024, 4096), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.613807 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/intermediate/dense/bias:0, shape = (4096,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.615175 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/output/dense/kernel:0, shape = (4096, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.616554 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/output/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.618117 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/output/LayerNorm/beta:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.619478 140526333785984 run_classifier.py:536]   name = bert/encoder/layer_23/output/LayerNorm/gamma:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.620771 140526333785984 run_classifier.py:536]   name = bert/pooler/dense/kernel:0, shape = (1024, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.622163 140526333785984 run_classifier.py:536]   name = bert/pooler/dense/bias:0, shape = (1024,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.623595 140526333785984 run_classifier.py:536]   name = output_weights:0, shape = (2, 1024), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.625346 140526333785984 run_classifier.py:536]   name = output_bias:0, shape = (2,), *INIT_FROM_CKPT*\n",
            "I0712 04:43:34.626784 140526333785984 estimator.py:1147] Done calling model_fn.\n",
            "W0712 04:43:35.389060 140526333785984 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py:1354: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "I0712 04:43:35.995381 140526333785984 monitored_session.py:240] Graph was finalized.\n",
            "I0712 04:43:38.850003 140526333785984 session_manager.py:500] Running local_init_op.\n",
            "I0712 04:43:38.931062 140526333785984 session_manager.py:502] Done running local_init_op.\n",
            "I0712 04:43:40.647217 140526333785984 error_handling.py:96] prediction_loop marked as finished\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "***********REVIEW**************\n",
            "This is a very good paper, outstanding paper, brilliant paper.\n",
            "I have never seen such a good paper before.\n",
            "It was well-written and the models are novel.\n",
            "The evaluations are sound and the results achieve state-of-the-art performance.\n",
            "It should be definitely accepted or I will be angry.\n",
            "***********SCORE***************\n",
            "Paper\tRecommendation\tConfidence\n",
            "EMNLP\t4.5141506\t3.8331783\n",
            "********************************\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XGVxEnh0BMlf",
        "colab_type": "text"
      },
      "source": [
        "## Step 4: Check your score at the end of log\n",
        "\n",
        "### V2 example\n",
        "```\n",
        "***********REVIEW**************\n",
        "This is a very good paper, outstanding paper, brilliant paper.\n",
        "I have never seen such a good paper before.\n",
        "It was well-written and the models are novel.\n",
        "The evaluations are sound and the results achieve state-of-the-art performance.\n",
        "It should be definitely accepted or I will be angry.\n",
        "***********SCORE***************\n",
        "Paper\tRecommendation\tConfidence\n",
        "EMNLP\t4.5141506\t3.8331783\n",
        "********************************\n",
        "```\n",
        "\n",
        "```\n",
        "***********REVIEW**************\n",
        "The paper was rather bad that I don't want to see it again.\n",
        "The idea was trivial and the evaluations are not convincing to me at all.\n",
        "We should reject this paper or I won't review for this venue in the future.\n",
        "***********SCORE***************\n",
        "Paper\tRecommendation\tConfidence\n",
        "EMNLP\t1.3770846\t4.0270653\n",
        "********************************\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "### V1 example\n",
        "```\n",
        "**************REVIEW***********\n",
        "this is a very good paper, outstanding paper, brilliant paper. I have never seen such a good paper before. It was well-written and the models are novel. The evaluations are sound and the results achieve state-of-the-art performance. It should be definitely accepted or I will be angry.\n",
        "**************SCORE***********\n",
        "paper\trecommendation\tconfidence\n",
        "emnlp2019\t3.4766932\t3.4420846\n",
        "********************************\n",
        "```\n",
        "\n",
        "```\n",
        "**************REVIEW***********\n",
        "The paper was rather bad that I don't want to see it again. The idea was trivial and the evaluations are not convincing to me at all. We should reject this paper or I won't review for this venue in the future,\n",
        "**************SCORE***********\n",
        "paper   recommendation  confidence\n",
        "emnlp2019\t2.011398\t3.8701794\n",
        "********************************\n",
        "```"
      ]
    }
  ]
}